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Abstract 

Systematic reviews have become increasingly critical to informing healthcare policy; however, they remain a time- 
consuming and labor-intensive activity. The extraction of data from constituent studies comprises a significant 
portion of this effort, an activity which is often needlessly duplicated, such as when attempting to update a 
previously conducted review or in reviews of overlapping topics. 

In order to address these inefficiencies, and to improve the speed and quality of healthcare policy- and decision- 
making, we have initiated the development of the Systematic Review Data Repository, an open collaborative Web- 
based repository of systematic review data. As envisioned, this resource would serve as both a central archive and 
data extraction tool, shared among and freely accessible to organizations producing systematic reviews worldwide. 
A suite of easy-to-use software tools with a Web frontend would enable researchers to seamlessly search for and 
incorporate previously deposited data into their own reviews, as well as contribute their own. 
In developing this resource, we identified a number of technical and non-technical challenges, as well as devised a 
number of potential solutions, including proposals for systems and software tools to assure data quality, stratify 
and control user access effectively and flexibly accommodate all manner of study data, as well as means by which 
to govern and foster adoption of this new resource. 

Herein we provide an account of the rationale and development of the Systematic Review Data Repository thus 
far, as well as outline its future trajectory. 
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Background 

Tasked with effectively integrating an ever-expanding 
sea of scientific data, findings from systematic reviews, 
which combine and synthesize data from many indivi- 
dual studies, have become invaluable to healthcare deci- 
sion-makers. In the US, the recent passage of the 
Patient Protection and Affordable Care Act has man- 
dated a renewed emphasis on evidence-based practice 
through comparative effectiveness research. Systematic 
reviews are therefore poised to play an even more pro- 
minent role in US health policy. However, conducting 
these reviews remains time-consuming and labor-inten- 
sive [1]. 

A fundamental step in conducting a systematic review 
is identifying relevant studies and extracting the essen- 
tial information from each study, information such as 
the study design, study population, description of the 
intervention and what it is being compared to and 



* Correspondence: jlau1@tuftsmedicalcenter.org 

Tufts Evidence-based Practice Center, Tufts Medical Center, Boston, MA 

02111, USA 



outcomes. Data across studies are then analyzed and 
synthesized. The extraction of one article alone may 
take several hours, and large reviews may include several 
hundred publications. Duplicate extractions by separate 
researchers are often conducted for verification as well, 
adding to resource needs and further drawing out the 
process. Therefore, data extraction is a major time-sink 
in conducting a systematic review. 

As systematic reviews conducted by different organiza- 
tions (serving different research purposes) often ask 
similar questions on the same topics, this process may 
be replicated numerous times for the same studies. The 
bodies of literature covered in such reviews frequently 
and necessarily overlap, and while the questions 
addressed vary, the data extracted from relevant publica- 
tions are largely identical, representing an enormous 
duplication of effort. Similarly, updates of existing sys- 
tematic reviews are commonly conducted to keep up 
with the latest scientific findings. As these updates are 
often conducted years after the original systematic 
review and often by different research groups, they fre- 
quently require re-extraction of previously reviewed 



Spri 



ringer 



© 2012 Ip et al; licensee BioMed Central Ltd. This is an Open Access article distributed under the terms of the Creative Commons 
Attribution License (http://creativecommons.0rg/licenses/by/2.O), which permits unrestricted use, distribution, and reproduction in 
any medium, provided the original work is properly cited. 



Ip et al. Systematic Reviews 2012, 1:15 
http://www.systematicreviewsjournal.eom/content/1 /I /1 5 



Page 2 of 7 



studies, as data from previous incarnations of a review 
are not always readily available. 

In June 2010, the Evidence-based Practice Center 
(EPC) at Tufts Medical Center, with support from the 
Agency for Healthcare Quality and Research (AHRQ), 
initiated development of the Systematic Review Data 
Repository (SRDR), a collaborative Web-based publicly 
available repository of systematic review data. As envi- 
sioned, this resource will reduce the unnecessary repli- 
cation of effort in conducting a systematic review, by 
serving as both a central archive and data extraction 
tool, shared among organizations producing systematic 
reviews worldwide. This database will be freely accessi- 
ble to facilitate evidence reviews and thus improve and 
speed policy-making with regards to healthcare. 

Main text 

Benefits of the Systematic Review Data Repository 

The anticipated benefits of the SRDR include: a more effi- 
cient means of producing and updating systematic reviews; 
improved access to more detailed information by consu- 
mers of review evidence; the promotion of transparency 
and reliability in the systematic review process via com- 
munal review; and enhanced cooperation and utility across 
related resources (such as the World Health Organiza- 
tions International Clinical Trials Registry Platform [2] or 
ClincialTrials.gov [3]). Such an archive will also facilitate 
novel avenues of research, such as investigations of 
research methodology (for example, comparing different 
types of outcomes across disciplines, or comparing metho- 
dological quality of studies based on funding sources), by 
concentrating relevant data into one readily available 
source. Several scenarios depicting the beneficial implica- 
tions of such a repository are outlined below. 

♦ Investigators search for data from existing systematic 
reviews that have been deposited in the repository. They 
then conduct a novel systematic review or update an 
existing one (or they decide to forego the effort comple- 
tely) depending on the results of their search and parti- 
cular research questions. 

♦ A user misinterprets and/or mistranscribes some infor- 
mation during data extraction into the repository. This 
discrepancy is identified and flagged for correction by the 
community (in a manner similar to various other colla- 
borative resources (in particular, see WikiProject Medicine 
[4]) and the data contributor is notified of the error. 

♦ A stakeholder interested in a particular topic 
searches the repository and quickly identifies related 
reviews through the repository's links to other external 
databases. 

Types of users 

To ensure that the deposited data are accurate and reli- 
able, we plan to allow only certified users to deposit and 



modify data. Although data in the SRDR system will be 
freely accessible to anyone via the Internet, a minimal 
set of procedural policies to certify data depositors will 
be instituted. We envision a system of tiered access to 
SRDR data and functionality. Based on an assessment of 
probable use cases, our proposed system contains four 
types of SRDR users, each with a specific set of data 
access permissions. The proposed user types in order of 
increasing privilege are: Viewer, Commentator, Contri- 
butor and Publisher/Editor. 
Viewer 

Any interested person would be allowed to view and 
download published data; however, no commenting, 
editing or supplementation of data would be permitted. 
Registration prior to accessing the SRDR would be 
voluntary. 
Commentator 

Any interested person with permission to post com- 
ments on published SRDR entries. Such users would 
have to register on the SRDR website, providing basic 
identifying and contact information, including their 
name, their affiliated organization and a valid email 
address. Following email verification, newly registered 
commentators would be required to accept the terms of 
use for the repository's discussion features. All com- 
ments would be vetted for clarity and adherence to sys- 
tem rules by the SRDR support team prior to posting. 
Contributor 

A registered individual who has been given permission 
to contribute data to a project housed in the repository. 
Accounts of this type would initially be assigned via cer- 
tified groups or organizations, rather than through the 
SRDR's Web registration. Initially, only EPCs and other 
invited groups would be granted rights to contribute 
data to the SRDR. Subsequently, additional contributors 
who wished to submit systematic reviews would be 
granted access under policies to be established with gui- 
dance from the SRDR governing body. 
Publisher/Editor 

Individuals granted the authority to publish (or make 
publicly viewable) a completed systematic review project 
on the SRDR Web site. It would be left to the partici- 
pating organizations to decide who and how many 
members of the organization would be designated for 
this role. 

Discussion 

Technical challenges 

To gain a thorough understanding of the issues to be 
considered in creating the SRDR, in June 2010 we con- 
vened a multidisciplinary panel of technical experts 
knowledgeable in systematic reviews and with expertise 
in trial registries, databases, systematic review methodol- 
ogies and/or biomedical informatics. Based on the 
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discussion during the expert panel meeting, a number of 
design requirements were established. First, the reposi- 
tory would have to be flexible and adaptable - with the 
ability to accommodate all manners of clinical research 
questions, studies and outcomes - in order to meet the 
needs and expectations of systematic reviewers from a 
variety of organizations and disciplines. Second, the 
repository should be easy to use and present a low hur- 
dle for users to contribute data. Ideally, it would be 
readily integrated into researchers' existing review pro- 
cesses, facilitating more efficient systematic reviews with 
minimal overhead. And third, the repository should be 
interoperable with existing systematic review databases 
and resources, leveraging universal and open standards 
for cross-communication and collaboration. 

The technical challenges in adhering to these design 
requirements include designing a system that balances 
the desired flexibility with easy-to-use editing, viewing 
and exporting capabilities for researchers. The amount 
and types of data that must necessarily be represented 
to accommodate the diverse needs of potential users are 
vast. Also, because the repository will rely largely on 
voluntary contribution, imposing rigid data formats or 
unintuitive interfaces would undermine the repository's 
adoption and potential value. However, such flexibility 
substantially complicates enabling data visualization, 
export (such as to statistical software) and search, in 
addition to adding complexity to the user-interface. 
While a feature-rich system is important to accommo- 
date diverse needs, ease of use remains critical to the 
project's ultimate success. Overall, the goal is to permit 
researchers of a range of technical expertise to effec- 
tively interact with the repository, both in contributing 
and retrieving data. 

Non-technical challenges 

We expect that the major hurdle to uptake of the 
SRDR will be overcoming the inertia of reviewers 
accustomed to their current routines, who may see lit- 
tle incentive to adopt new methods. In addition to pro- 
viding compelling functionality that will help facilitate 
systematic reviews, the necessary encouragement may 
need to come from funders of systematic reviews, who 
are driven by the need for rapid production of reviews 
and have long-term interest in the repository's poten- 
tial to drive down costs. The provision of strong and 
convenient technical support should also help alleviate 
user reluctance. The issue of custodianship, including 
curatorial responsibilities to assure data accuracy, will 
also need to be worked out. Whether the repository 
will eventually be a public or private, US or interna- 
tional entity will require careful deliberation by all 
involved parties. 



Governance structure 

Other, non-technical challenges that remain to be over- 
come primarily concern governance and data quality 
assurance processes. In further discussion with the 
expert panel, we determined that it is important to 
establish a governing or advisory body for this data 
repository. The SRDR governing body would be charged 
with setting overall strategic goals and priorities, and 
establishing and managing policies and processes, 
including those related to data quality control and user 
certification. We believe that a joint committee of var- 
ious stakeholders would be best suited to guide develop- 
ment of the repository, by guaranteeing representation 
of the needs of the SRDR's various constituencies. 

We believe that the structure of the proposed govern- 
ing body will differ somewhat based on the eventual 
permanent funding source. Three possible scenarios for 
this source (and therefore the governing structure) are 
that the SRDR will continue to be fully funded and 
managed by the US government; that the SRDR will be 
funded and operated as an independent non-govern- 
mental entity, either through a stand-alone organization 
(such as the Patient Centered Outcomes Research Insti- 
tute) or as a unit within an academic institution; or that 
the SRDR will operate as a commercial or fee-for-service 
resource. We believe the last option is undesirable 
because it runs counter to the SRDR's stated purpose to 
provide an open and freely- accessible repository to pro- 
mote systematic review research. Therefore, the last sce- 
nario is not discussed further. 

As previously mentioned, a governing body will over- 
see the management of the SRDR. Representation will 
likely include members from EPCs, the National Library 
of Medicine, the biomedical informatics community, 
clinical practice guideline producing entities, interna- 
tional stakeholders such as the Cochrane Collaboration 
and the Center for Reviews and Dissemination in the 
UK and other relevant research organizations and user 
communities. Should it continue to be funded by the 
US government, the responsible agency official would 
select and invite the advisory committee members fol- 
lowing consultation with stakeholders, or oversee the 
contracting organization's selection process, and would 
determine the appropriate size of the committee and its 
terms of service. 

Should the SRDR operate as an independent non-gov- 
ernmental entity, it may reside within an academic insti- 
tution or exist as a standalone resource. Financial 
support would likely be provided in a non-commercial 
context, such as via public or private grants, donations 
and/or other similar funding mechanisms. The 
Cochrane collaboration might serve as an organizational 
model (see Cochrane Policy Manual [5]) for such an 
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independent entity. As in the Cochrane collaboration, 
the SRDR's mission would be accomplished by groups 
of volunteers. Members of relevant stakeholder groups 
would come together because they share an interest in 
reliable, up-to-date evidence in health care. 
Data quality control 

It is essential for the SRDR's success that the deposited 
data are as accurate as possible. We believe that a multi- 
layered approach to data quality control, applied both 
pre- and post-deposit, would best ensure that the SRDR 
accumulates only quality data. During data entry, we 
intend to implement per-field error checking (for exam- 
ple, built-in range checks for predefined numerical data 
types). Some error checking may also be user-specified 
(for example, range and direction for numeric data 
fields). In addition, process and data visualization (for 
example, progress meters, summary table creation and 
export) would assist users in entering data accurately 
during the data extraction process. Process and data 
visualization would allow project leads to review sum- 
mary data in order to identify errors or missing data 
during the production of the systematic review. 

For published data (those made publicly viewable), we 
propose two approaches to facilitate quality assurance: 
regular data audits and wiki-like editorial features for 
flagging or commenting on deposited data. We antici- 
pate that the SRDR will be archiving data from a large 
number of systematic reviews. Thus, centralized review 
and curation of all deposited records will quickly 
become infeasible. Regular data audits of randomly 
selected records would strike a reasonable balance 
between maintaining data quality and ensuring the qual- 
ity assurance workload remains feasible. These audits 
could initially be conducted by the Tufts EPC (as the 
group most familiar with the repository) and then, going 
forward, by each certified organization's resident SRDR 
coordinator. Public collaboration has been identified as 
potentially useful in a number of scenarios applicable to 
the systematic review activities sponsored by AHRQ. By 
enabling collaborative commentating and data review, 
the entire user-base can be enlisted in the quality assur- 
ance process. 

Potential for error or misuse 

Regardless of the measures outlined above, the SRDR 
will inevitably engender erroneous data or misuse. Such 
incidents may range from simple typographical errors in 
deposited records to more insidious mistakes of misin- 
terpretation or misapplication of archived data. It is 
worth noting, however, that such errors are not 
restricted to the SRDR but are endemic to the creation 
of all systematic reviews, with the important distinction 
that the SRDR provides a structured mechanism for 
external review and correction of these errors while tra- 
ditional methods generally do not. 



Rather than erroneous interpretation, the most likely 
negative outcome to emerge from the SRDR would be 
abuse of the good faith in which the SRDR tools are 
provided. SRDR users could potentially take advantage 
of the repository's online extraction and analysis tools 
without eventually making their data publically available. 
It is our hope, however, that the utility the SRDR pro- 
vides and the wider benefits to the research community 
it engenders would be obvious and compelling enough 
to encourage a mutually beneficial reciprocity. 

Conclusions 

Current status 

As of January 2012, we have developed an initial version 
of the SRDR system. AHRQ plans to host the server 
and database for storing the data extracted into the 
SRDR. The current version allows contributors to initi- 
ate new systematic review projects, create extraction 
forms and extract data into the repository using a series 
of flexible interfaces. The underlying database was 
designed in a way such that the data, once entered into 
the system, will be easily retrieved and formatted into 
customizable summary tables for publication. 

At this time, data extractors within the Tufts EPC 
have been testing SRDR functionality, yielding continu- 
ing improvements. Selected screenshots of the current 
implementation of the SRDR are depicted in Figures 1, 
2, 3, 4 and 5. The focus of the archive is on enlisting 
future data collection, as we do not anticipate that many 
investigators will enter previously extracted data. How- 
ever, we are creating import tools to simplify copying 
data extracted in other databases into the SRDR. This 
will enable those who wish to continue using their pre- 
ferred software for data extraction to easily contribute 
previously extracted data into the SRDR archive. 

Additional functionality currently under development 
prior to deployment includes a data comparison tool for 
use in double data extraction, a summary table creator 
and linking the SRDR to other databases (for example, 
ClinicalTrials.gov [3]). The next stage of deployment is 
scheduled for June 2012, when the SRDR will be opened 
to other AHRQ-funded EPCs, with general availability 
to be provided shortly thereafter. 

Next steps 

Notably, there exist a number of registries or comple- 
mentary resources with similar ethos, including the 
aforementioned International Clinical Trials Registry 
Platform [2] and ClinicalTrials.gov [3,6], the Human 
Studies Database [7,8], the Guidelines International Net- 
work International Guidelines Database [9-11] and the 
Centre for Reviews and Dissemination's international 
register of ongoing systematic reviews [12,13]. Interoper- 
ability among these organizations and resources would 
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Exit Full Screen SRDR Home My Projects Published Projects Extraction Forms Feedback Help Logged in as admin Log Out 

Editing Project: Prostate Cancer 2009 (Project status: complete. Published) 

Project Information 

Use this form to modify your project, then dick the "Save and Continue' button to follow the process shown at the right 



Project Title |p ros tate Cancer 2009 



Project 
Description 



Funding 
Source 



Project 
Notes 



Key 

Questions 



ransferring data from bioportal.bioontology.org. 



Background: Radiation therapy is one of many 
treatment options for patients with prostate cancer. 



Purpose: To update findings about the clinical and v 
biochemical outcomes of radiation therapies for 
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Question Text 



1 What are the benefits and harms of radiation therapy for 
clinically localized prostate cancer compared to watchful 
waiting (including active surveillance) in terms of clinical 
outcomes'' 



2 What are the benefits and harms of different forms of 
radiation therapy for clinically localized prostate cancer in 
terms of clinical outcomes? The comparisons of interest are 
between the following radiation modalities stereotactic body 
radiation therapy (SBRT. including CyberKnife therapy), 
classically fractionated external beam radiation therapy 
(EBRT. including 3D-conformal radiation therapy intensity 



Figure 1 Partial view of the Project Information Section of a Systematic Review Data Repository data extraction form. 



be of mutual benefit. Our intent is not to create a one- 
size-fits-all tool, or to supplant existing ones, but rather 
to improve, complement and extend existing research 
infrastructure. While several other commercial and pro- 
fessional ventures, such as Doctor Evidence [14] and the 
American Dietetic Association Evidence Analysis Library 
[15,16], have undertaken extracting data from published 
studies into dedicated databases, these endeavors remain 
proprietary and restricted resources. The SRDR is 
designed to be an open-access and freely available sys- 
tem. To our knowledge, this proposed archive is unique 



in its scope and mission, and will provide a tool sorely 
lacking in the research community. 

Over the next phase of development, the Tufts EPC 
will continue to refine the SRDR and expand its features 
and functionality. We have formed a new expert panel 
to solicit further feedback as the repository becomes 
more widely used and to seek a greater range of stake- 
holder input. In addition to the relevant federal agen- 
cies, we have representatives from other EPCs, the 
Cochrane Collaboration, organizations that produce 
guidelines, as well as other relevant health care 
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Study Design 
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2. Multi-Centered? 

©Yes 
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4. Study Design 
|RCT 



• population 300.000.000 
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□ Africa 

□ Australia 

6. Trial or Cohort Name Inclusion 



Figure 2 Partial view of the Study Enrollment and Design Information Entry Section. 
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Figure 3 Partial view of the Outcome Data Entry Section. 
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Figure 4 Partial view of the Study Quality Appraisal Entry Section. 



Dimension Value Notes Instructions 


Appropriate Randomization Technique (y/n/noVNA) 


|Yes 


H 1 1 


Alocation concealment [Y, N. Nd, NA] 
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Appropriate Washout Period (y/n/nd/NA) 
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Intention to treat analysis [Y, N, Nd] 


|Yes 


il 1 1 




Appropriate statistical analysis [Y. N] 
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Editing Project: Prostate Cancer 2009 (Project status: complete. Publ ished) 

Project Summary 

Title and Description 

| Prostate Cancer 2009 

d Quality, U S Department of Health and Human Services (Contract No 290 2007 



What are the benefits and harms of different forms of radiation therapy for clinically localized prostate cancer in terms of clinical 
outcomes' The comparisons of interest are between the following radiation modalities stereotactic body radiation therapy (SBRT 
including CyberKnife therapy) classically fractionated external beam radiation therapy (EBRT including 3D-conformal radiation therapy 
intensity modulated radiation therapy and particle therapy) high dose rate brachytherapy (HDR) and low dose rate brachytherapy 
(LDR) These modalities will be specifically compared with each other i e SBRT vs EBRT SBRT vs HDR SBRT vs LDR EBRT vs 
HDR. EBRT vs LDR. and HDR VS LDR 

How do specific patient characteristics e g age race/ethnicity presence or absence of comorbidities preferences (e g . tradeoff of 
treatment-related adverse effects vs potential for disease progression) affect the outcomes of these different forms of radiation 
therapy'' 




I 



Collaborators 



SRDR Administrator 




Figure 5 Partial view of the Project Description and Key Questions Section 
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organizations, such as Kaiser- Permanente, to assist with 
further policy development. Many of the aforementioned 
organizations are working on databases for other kinds 
of information related to systematic reviews, including 
abstracts of systematic reviews and summaries of pri- 
mary studies. It is our hope that by engaging these 
other organizations and the wider stakeholder commu- 
nity - patients, payers, researchers and governments 
alike - we can foster a collaborative environment that 
will enable this repository to grow into a valuable and 
permanent resource of benefit to all. 
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